Image processing system, mobile object, image processing method, and storage medium

ABSTRACT

An image processing system includes: an image acquisition unit configured to acquire image data generated by an imaging device that captures an optical image having a low-distortion region and a high-distortion region; a first image recognition unit configured to perform image recognition on image data of at least a partial region in the image data acquired by the image acquisition unit to output a first image recognition result; a second image recognition unit configured to perform image recognition on image data of a region wider than the partial region in the image data acquired by the image acquisition unit to output a second image recognition result; and an integration processing unit configured to output an image recognition result integrated on the basis of the first image recognition result and the second image recognition result.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing system, a mobileobject, an image processing method, and a storage medium for performingimage recognition.

Description of the Related Art

In recent years, there has been a demand for replacing a rear-viewmirror (back view mirror) mounted in a vehicle with an electronic rearview mirror. Japanese Patent Laid-Open No. 2010-95202 discloses anelectronic rear view mirror system that includes an imaging unit adaptedto image a rear side outside a vehicle as an imaging range and a displayunit disposed inside the vehicle and enables a driver to check rearconditions outside the vehicle through display of an image captured bythe imaging unit on a display inside the vehicle.

On the other hand, there is a rear side checking system that enables adriver to check a blind angle behind a vehicle when the vehicle movesbackward or the like. Japanese Patent Laid-Open No. 2004-345554discloses a rear side checking system that includes a camera placed toimage a side behind a vehicle and is for enabling a driver to check ablind angle behind the vehicle at the time of backward moving throughdisplay of a captured image inside a vehicle interior.

The imaging unit adapted to capture an image for the electronic rearview mirror as described above is required to have high resolution forallowing the driver to check rear conditions that are relatively farfrom the driver with more accuracy. On the other hand, the camera forthe rear side checking system is required to image a wider range inorder to check safety in a wider range including blind angles and rearlateral sides behind the vehicle to avoid collision at the time ofbackward moving.

Therefore, if the electronic rear view mirror system and the rear sidechecking system are mounted in a vehicle at the same time, and thecamera for the electronic rear view mirror system and the camera for therear side checking system are individually mounted, an in-vehicle imageprocessing system becomes complicated. Such a problem similarly occursin an automated driving system that performs automated driving and thelike by disposing a plurality of cameras to image conditions in thesurroundings of a vehicle.

Although it is possible to reduce the number of cameras to be placed ina vehicle by adopting cameras using super-wide-angle lenses such asfisheye lenses to address this problem, it is still difficult to obtainimages with a high resolution while it is possible to obtain a wideangle of view. If image recognition is performed in the system usingfisheye lenses, and the image recognition is carried out without anydistortion correction, there is a likelihood that recognition accuracyin a periphery portion of the image may be degraded.

On the other hand, if the image recognition is carried out afterdistortion correction, there is a likelihood that a processing load maybecome large and it may take a long time depending on an image size orresolution.

Moreover, if the distortion correction is performed on a plurality ofimages captured by a plurality of cameras, there is a problem that theprocessing load may become yet larger.

SUMMARY OF THE INVENTION

An image processing system according to an aspect of the presentinvention comprises at least one processor or circuit configured tofunction as:

-   an image acquisition unit configured to acquire image data generated    by an imaging device that captures an optical image including a    low-distortion region and a high-distortion region;-   a first image recognition unit configured to perform image    recognition on image data of at least a partial region in the image    data acquired by the image acquisition unit to output a first image    recognition result;-   a second image recognition unit configured to perform image    recognition on image data of a region wider than the partial region    in the image data acquired by the image acquisition unit to output a    second image recognition result; and-   an integration processing unit configured to output an image    recognition result integrated on the basis of the first image    recognition result and the second image recognition result.

Further features of the present invention will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for explaining positional relationships between avehicle 1 and camera units 11 to 14 according to a first embodiment.

FIG. 2A and FIG. 2B are diagrams for explaining optical properties ofthe camera units 11 to 14.

FIG. 3 is a block diagram for explaining a configuration of an imageprocessing system 100 according to the first embodiment.

FIG. 4 is a flowchart for explaining a processing flow of cameraprocessing units 31 to 34.

FIG. 5 is a flowchart for explaining a processing flow of an integrationprocessing unit 40.

FIG. 6 is a flowchart for explaining an integration processing flow ofthe integration processing unit 40.

FIG. 7 is a flowchart for explaining an example of an image displaymethod according to the first embodiment.

FIGS. 8A to 8E are diagrams for explaining a relationship of ahigh-resolution region, a low-resolution region, and a plurality oftypes of display regions according to the first embodiment.

FIG. 9A is a diagram for explaining a display example of an image with areference angle of view, FIG. 9B is a diagram for explaining a displayexample of an image with a narrow angle of view, and FIG. 9C is adiagram for explaining a display example of an image with a wide angleof view.

FIG. 10A is a diagram for explaining a display example of a displayscreen 501 of a first display unit 50, and FIG. 10B is a diagram forexplaining a display example of a display screen 511 of a second displayunit 51.

FIG. 11A is a diagram for explaining an example of an image 85B with anarrow angle of view at the time of backward moving, and FIG. 11B is adiagram illustrating an example of an image 86B with a wide angle ofview for backward traveling.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the accompanying drawings, favorablemodes of the present invention will be described using Embodiments. Ineach diagram, the same reference signs are applied to the same membersor elements, and duplicate description will be omitted or simplified.

First Embodiment

In a first embodiment, an improved method for balancing, with a smallnumber of cameras, high-definition display for an electronic rear viewmirror and wide-range display for checking surroundings of a vehiclesuch as a rear side will be described.

FIG. 1 is a diagram for explaining a positional relationship between avehicle 1 (an automobile, for example) and camera units 11 to 14 in thefirst embodiment. The vehicle 1 operates as a mobile object, and all thecamera units 11 to 14 operate as imaging devices.

In the first embodiment, the camera units 11, 12, 13, and 14 areinstalled on the front side, the right side, the rear side, and the leftside of the vehicle 1 that is a mobile object, respectively, asillustrated in FIG. 1 . Note that although the image processing systemincludes the four camera units in the first embodiment, the number ofcamera units that the image processing system has is not limited tofour. The image processing system may be any image processing systemincluding one or more camera units.

The camera units 11 to 14 are installed to image the side in front, theright side, the left side, and the rear side of the vehicle 1 that is amobile object, as imaging ranges, respectively. Each of the camera units11 to 14 has substantially similar components. For example, each of thecamera units 11 to 14 includes an imaging device that captures anoptical image and an optical system that generates an optical image on alight receiving surface of the imaging device. For example, the opticalsystem that each of the camera units 11 to 14 has is installed such thatan optical axis thereof is substantially horizontal.

The optical systems that each of the camera units 11 to 14 has areconfigured to be able to obtain a high-definition image at a narrowangle of view in the surroundings of the optical axis and obtain acaptured image with a low resolution at a wide angle of view. Note thatin FIGS. 1, 11 a to 14 a illustrate imaging angles of view at whichhigh-resolution images can be captured while 11 b to 14 b illustrateimaging angles of view at which low-resolution images can be captured.

Next, optical properties of the optical systems that the camera units 11to 14 have will be described with reference to FIG. 2A and FIG. 2B.Although the optical properties of the optical systems that the cameraunits 11 to 14 have may not be the same, it is assumed that the opticalproperties of the optical systems that the camera units 11 to 14 haveare substantially the same in the first embodiment. Therefore, opticalproperties of the optical system that the camera unit 11 has will bedescribed as examples in FIG. 2A and FIG. 2B.

FIG. 2A is a diagram illustrating, as a contour line, an image height yat each half angle of view on the light receiving surface of the imagingdevice of the optical system that the camera unit 11 has. FIG. 2B is adiagram representing projection properties representing a relationshipbetween the image height y of the optical system that the camera unit 11has and a half angle of view θ. FIG. 2B is illustrated with the halfangle of view (an angle formed by an optical axis and an incident lightbeam) θ shown as the horizontal axis and with the image formation height(image height) y on the sensor plane (on the image plane) of the cameraunit 11 defined by the vertical axis.

The optical system that the camera unit 11 has is configured such thatthe projection properties y(θ) differ in a region of less than apredetermined half angle of view θa and in a region of equal to orgreater than the half angle of view θa as illustrated in FIG. 2B.Therefore, resolutions differ depending on regions when the amount ofincrease in image height y with respect to the half angle of view θ perunit is defined as a resolution.

It is also possible to state that the local resolution is represented bya differential value dy(θ)/dθ at the half angle of view θ of theprojection property y(θ). In other words, it is possible to state thatthe larger the inclination of the projection property y(θ) in FIG. 2Bis, the higher the resolution is. Also, it is possible to state that thelarger the interval of the image height y at each half angle of view asa contour in FIG. 2A is, the higher the resolution is.

In the first embodiment, a region near the center that is generated onthe sensor plane when the half angle of view θ is less than thepredetermined half angle of view θa is referred to as a high-resolutionregion 10 a, and a region near the outside where the half angle of viewθ is equal to or greater than the predetermined half angle of view θa isreferred to as a low-resolution region 10 b. Note that the angle of viewin the high-resolution region 10 a corresponds to the imaging angle ofview 11 a, while the angle of view in the low-resolution region 10 bcorresponds to the imaging angle of view 11 b.

Note that in the first embodiment, the high-resolution region 10 a is alow-distortion region where the amount of distortion is relativelysmall, while the low-resolution region 10 b is a high-distortion regionwhere the amount of distortion is relatively large. Therefore, thehigh-resolution region and the low-resolution region may be referred toas a low-distortion region and the high-distortion region, respectively,in the first embodiment.

The optical system that the camera unit 11 has is configured such thatthe projection property y(θ) in the high-resolution region(low-distortion region) 10 a is greater than f × θ (f is a focaldistance of the optical system that the camera unit 11 has). Also, theprojection property y(θ) in the high-resolution region (low-distortionregion) is set to be different from the projection property in thelow-resolution region (high-distortion region).

When θmax is defined as a maximum half angle of view that the opticalsystem of the camera unit 11 has, it is desirable that a ratio θa/θmaxbetween θa and θmax be equal to or greater than a predetermined lowerlimit value, and for example, it is desirable that the predeterminedlower limit value be 0.15 to 0.16.

Also, it is desirable that the ratio θa/θmax between θa and θmax beequal to or less than a predetermined upper limit value and be 0.25 to0.35, for example. If θa is set to 90°, the predetermined lower limitvalue is set to 0.15, and the predetermined upper limit value is set to0.35, for example, it is desirable to determine θa within a range of13.5 to 31.5°.

Moreover, the optical system that the camera unit 11 has is configuredsuch that the projection property y(θ) thereof satisfies the followingexpression as well:

$1 < \frac{f \times \mspace{6mu} s\mspace{6mu} i\mspace{6mu} n\theta_{\max}}{y\left( \theta_{\max} \right)} \leq A$

Here, f is a focal distance of the optical system that the camera unit11 has as described above, and A is a predetermined constant. It ispossible to obtain a higher center resolution than that of a fisheyelens based on an orthographic projection scheme (y = f × sinθ) havingthe same maximum image formation height by setting the lower limit valueto one, and it is possible to maintain satisfactory optical performancewhile obtaining an angle of view equivalent to that of the fisheye lensby setting the upper limit value to A. It is only necessary to determinethe predetermined constant A in consideration of a balance betweenresolutions in the high-resolution region and the low-resolution region,and it is desirable that the predetermined constant A be 1.4 to 1.9.

It is possible to obtain a high resolution in the high-resolution region10 a and to reduce the amount of increase in image height y with respectto the half angle of view θ per unit and to image a wider angle of viewin the low-resolution region 10 b by configuring the optical system asdescribed above. Therefore, it is possible to obtain a high resolutionin the high-resolution region 10 a while setting the wide angle of viewthat is equivalent to that of the fisheye lens as an imaging range.

In the first embodiment, properties that are close to those of thecenter projection method (y = f × tanθ) and the equidistant projectionmethod (y = f × θ) that are projection properties of an optical systemfor normal imaging are set in the high-resolution region (low-distortionregion), and it is thus possible to obtain low optical distortion andthus high-definition display. Therefore, it is possible to obtain anatural perspective when surrounding vehicles such as a vehicle aheadand a following vehicle and the like are visually recognized and toobtain satisfactory visibility while curbing degradation of imagequality.

Note that since it is possible to obtain similar effects by anyprojection property y(θ) as long as it satisfies the aforementionedcondition of Expression 1, the first embodiment is not limited to theprojection properties illustrated in FIG. 2 . Note that in the firstembodiment, the optical system having the projection property y(θ) thatsatisfies the aforementioned condition of Expression 1 may be referredto as a different-angle-of-view lens.

Note that the angles of view of the high-resolution regions 10 a of theoptical systems that the camera units 11 to 14 have correspond toimaging angles of view 11 a to 14 a, respectively, and the angles ofview of the low-resolution regions 10 b of the optical systems that thecamera units 11 to 14 have correspond to imaging angles of view 11 b to14 b, respectively.

Next, a configuration of the image processing system 100 in the firstembodiment will be described with reference to FIG. 3 . FIG. 3 is ablock diagram for explaining the configuration of the image processingsystem 100 in the first embodiment.

In FIG. 3 , the image processing system 100 is mounted in the vehicle 1.Each of the camera units 11 to 14 includes any of imaging units 21 to 24and any of camera processing units 31 to 34. Each of the imaging units21 to 24 includes any of different-angle-of-view lenses 21 c to 24 c andany of imaging devices 21 d to 24 d.

The imaging devices 21 d to 24 d include, for example, CMOS imagesensors or CCD image sensors. Here, each of the imaging units 21 to 24functions as an image acquisition unit and acquires image data from anyof the imaging devices 21 d to 24 d that capture optical imagesincluding a low-distortion region and a high-distortion region.

Each of the different-angle-of-view lenses 21 c to 24 c that are opticalsystems is configured of one or more optical lenses, has the projectionproperty y(θ) that satisfies the aforementioned condition of Expression1, and generates an optical image including a low-distortion region anda high-distortion region on a light receiving surface of any of theimaging devices 21 d to 24 d. Each of the imaging devices 21 d to 24 dperforms photoelectrical conversion on the optical image and outputsimaging data.

RGB color filters, for example, are aligned for each pixel on each ofthe light receiving surfaces of the imaging devices 21 d to 24 d. TheRGB alignment is, for example, Bayer alignment. Therefore, each of theimaging devices 21 d to 24 d is configured such that R, G, R, G pixeldata is successively output from a predetermined row and G, B, G, Bpixel data is successively output from the next row, for example, inaccordance with the Bayer alignment.

Each of the camera processing units 31 to 34 is accommodated along withany of the imaging units 21 to 24 in a casing of any of the camera units11 to 14. Each of the camera processing units 31 to 34 processes theimaging data output from any of the imaging units 21 to 24.

Each of the camera processing units 31 to 34 includes any of imageprocessing units 31 a to 34 a, any of recognition units 31 b to 34 b,and any of camera information units 31 c to 34 c. The image processingunits 31 a to 34 a perform image processing on imaging data output fromthe imaging units 21 to 24, respectively. Note that a part or entiretyof the camera processing unit 31 may be implemented by a laminatedsignal processing unit inside the imaging devices 21 d to 24 d.

For example, each of the image processing units 31 a to 34 a performsDe-Bayer processing on image data input from any of the imaging units 21to 24 in accordance with the Bayer alignment and converts the image datainto image data based on an RGB luster format. Moreover, each of theimage processing units 31 a to 34 a may perform white balanceadjustment, gain/offset adjustment, gamma processing, color matrixprocessing, lossless compression processing, and the like.

Each of the recognition units 31 b to 34 b performs image recognition ona predetermined object (for example, an automobile, a person, anobstacle, or the like) from image data before distortion correctionafter image processing performed by any of the image processing units 31a to 34 a. For example, each of the recognition units 31 b to 34 bperforms image recognition on image data corresponding to alow-distortion region in a state of image data before distortioncorrection without performing distortion correction thereon and outputsa first image recognition result.

Note that each of the recognition units 31 b to 34 b performs imagerecognition processing on image data before distortion correctionobtained at least from the high-resolution region 10 a and recognizesthe predetermined target. Therefore, the image recognition processingmay be performed after only the high-resolution region 10 a isextracted.

In such a case, each of the recognition units 31 b to 34 b may alsoperform image recognition processing on image data before distortioncorrection obtained from the low-resolution region 10 b. However, sincedistortion correction has not been performed on the image data beforedistortion correction, the image at the periphery portion of thedifferent-angle-of-view lens has large distortion, and reliability ofthe recognition is degraded.

Note that each of the recognition units 31 b to 34 b may extract theimage data before distortion correction obtained from thehigh-resolution region 10 a and perform the image recognition processingonly on the image data before distortion correction obtained from thehigh-resolution region 10 a. In such a case, it is desirable that theextracted region have a rectangular shape for the image recognitionprocessing.

If the extracted region has a rectangular shape, the extracted regionmay be only a part inside the high-resolution region 10 a (for example,a rectangular shape that is inscribed in the high-resolution region 10a) or may be a rectangle including both the high-resolution region 10 aand the low-resolution region 10 b.

Here, the recognition units 31 b to 34 b function as first imagerecognition units that perform image recognition in image data in atleast partial regions out of image data acquired by the imageacquisition units and output first image recognition results. Note thatin the first embodiment, the partial regions are regions correspondingto low-distortion regions.

Each of the recognition units 31 b to 34 b transmits a set of a type andcoordinates of an object as a recognition result to an integrationprocessing unit 40. On the other hand, each of the recognition units 31b to 34 b receives prediction information that is a set of the type ofthe object and information regarding a moving direction of the object orpriority recognition region information from an integration control unit41 c of the integration processing unit 40. The prediction informationwill be described later.

In the first embodiment, an output of the recognition unit 31 b of thecamera unit 11 installed on the front side is also supplied directly toa traveling control unit (ECU) 60. This is because there may be a casein which it is necessary to immediately stop traveling on the basis of aresult of recognizing an obstacle or the like obtained by therecognition unit 31 b or it is necessary to control traveling to avoidan obstacle.

Each of the camera information units 31 c to 34 c holds camerainformation of any of the camera units 11 to 14 in a memory (such as aROM) in advance. Each piece of camera information includes opticalproperties of any of the different-angle-of-view lenses 21 c to 24 c,the number of pixels, gamma properties, sensitivity properties, a framerate, and attachment coordinates and posture information in vehiclecoordinates of the camera units of any of the imaging devices 21 d to 24d.

Moreover, each piece of camera information may include informationregarding an image processing method and an image format when image databefore distortion correction is generated by any of the image processingunits 31 a to 34 a. Note that the attachment coordinates and the posturecoordinates are relative coordinates with respect to the vehicle 1.Also, each piece of camera information is information unique to any ofthe imaging units 21 to 24 and is different from each other, and theinformation is transmitted to the integration processing unit 40 and isreferred to when the integration processing unit 40 performs imageprocessing.

Note that a CPU that serves as a computer and a memory that serves as astorage medium and stores a computer program are incorporated insideeach of the camera processing units 31 to 34. Also, each CPU isconfigured to control each component included in each of the cameraprocessing units 31 to 34 by executing the computer program in thememory.

Note that in the first embodiment, the image processing units 31 a to 34a and the recognition units 31 b to 34 b use, for example, hardware suchas dedicated circuits (ASICs) or processors (reconfigurable processors,DSPs, graphics processing units (GPUs)), for example. It is thuspossible to realize an increase in speed for image recognition in thehigh-resolution region and to increase the likelihood that accidents canbe avoided.

The integration processing unit 40 includes a system on chip (SOC)/fieldprogrammable gate array (FPGA) 41, a CPU 42 that serves as a computer,and a memory 43 that serves as a storage medium.

The CPU 42 performs various kinds of control on the entire imageprocessing system 100 by executing the computer program stored in thememory 43. Note that in the first embodiment, the integration processingunit 40 is accommodated in a casing that is separated from those for thecamera units.

The SOC/FPGA 41 includes an image processing unit 41 a, a recognitionunit 41 b, and an integration control unit 41 c. The image processingunit 41 a acquires image data before distortion correction from each ofthe camera processing units 31 to 34 and acquires camera information ofeach of the camera units 11 to 14 from each of the camera informationunits 31 c to 34 c.

Each piece of camera information includes optical properties of thedifferent-angle-of-view lenses 21 c to 24 c, the numbers of pixels, thephotoelectric conversion properties, γ properties, sensitivityproperties, format information of image data before distortioncorrection of the imaging devices 21 d to 24 d as described above. Also,each piece of camera information includes the attachment coordinates andthe posture information in the vehicle coordinates of the camera units.

The image processing unit 41 a performs distortion correction in theimage data obtained from the low-resolution region 10 b of each of theimaging units 21 to 24. In the first embodiment, the image processingunit 41 a does not perform distortion correction since the image dataobtained from the high-resolution region 10 a includes substantially nodistortion.

However, the image processing unit 41 a may also perform simplifieddistortion correction on the image data obtained from thehigh-resolution region 10 a. The image processing unit 41 a synthesizesthe image data in the low-resolution region 10 b with the image data inthe high-resolution region 10 a of each of the imaging units 21 to 24after the distortion correction such that they are smoothly connected,and thereby generates an overall image for each of the imaging units 21to 24.

Note that if distortion correction is performed on both the image datain the low-resolution region 10 b and the image data obtained from thehigh-resolution region 10 a, the image data before the distortioncorrection obtained from each of the image processing units 31 a to 34 amay be directly subjected to the distortion correction.

The recognition unit 41 b performs image recognition processing on theoverall image (including the peripheral portions of the image) for eachof the imaging units 21 to 24 after performing the distortion correctionon at least the low-resolution region and performs image recognition fora predetermined object (for example, an automobile, a person, or anobstacle) in the overall image for each of the imaging units 21 to 24.For example, the recognition unit 41 b performs the distortioncorrection at least on the image data corresponding to thelow-resolution region (high-distortion region), then performs imagerecognition thereon, and outputs a second image recognition result.

Note that the recognition unit 41 b may refer to recognition results(the type and the coordinates of the target) of the recognition units 31b to 34 b if image recognition processing is to be performed on theoverall image (including the peripheral portions of the image). Here,the recognition unit 41 b functions as a second image recognition unitthat performs image recognition on image data in a wider region than thepartial region on which the image recognition has been performed by thefirst image recognition unit, out of the image data acquired by theimage acquisition unit and outputs a second image recognition result.

The second image recognition unit performs image recognition on both theimage data corresponding to the high-resolution region 10 a that is thelow-distortion region and the low-resolution region 10 b that is thehigh-distortion region and outputs the second image recognition result.

The image processing unit 41 a may synthesize images of the plurality ofimaging units such that they are connected to each other and generate apanoramic synthesized image. In such a case, it is desirable that atleast a part of the imaging angle of view of each of the images of theplurality of imaging units that are connected to each other be set tohave a mutually overlapping region of an amount that is equal to orgreater than a predetermined amount.

The recognition unit 41 b may perform image recognition on the panoramicsynthesized image. In this manner, it is possible to perform imagerecognition of an object that is imaged across angles of view of theplurality of imaging units, for example. This is because there may be acase in which substantially the entire object appears in the panoramicsynthesized image and it is possible to perform image recognition in theimage processing although there is a case in which it is not possible toknow the overall picture of the object from the individual entire imagesfrom the imaging units.

The integration control unit 41 c outputs an integrated imagerecognition result by adopting a recognition result with higherreliability if the recognition results of the recognition units 31 b to34 b are different from the recognition result of the recognition unit41 b, for example.

For example, proportions of the object occupying the images recognizedby the recognition units 31 b to 34 b may be compared with theproportion of the same object occupying the image recognized by therecognition unit 41 b, and the recognition result with a higherproportion may be determined and adopted as having higher reliability.

Alternatively, in a case of an object which lies across both thehigh-resolution region and the low-resolution region, a recognitionresult of the recognition unit 41 b may be determined and adopted ashaving higher reliability rather than the recognition results of therecognition units 31 b to 34 b.

Alternatively, if the position of the object recognized by therecognition units 31 b to 34 b is at a peripheral part of the image, itmay be determined that reliability is low, and the recognition result ofthe recognition unit 41 b may be determined and adopted as having higherreliability.

Alternatively, the recognition unit 41 b may perform image recognitiononly on the low-resolution region in a state in which distortioncorrection has been performed on the low-resolution region, and if thereis an object which lies across the low-resolution region and thehigh-resolution region, image recognition may be performed on thetarget. For example, control may be performed such that the recognitionunit 41 b does not perform image recognition processing on theassumption that reliability of the recognition performed by therecognition units 31 b to 34 b is higher for the object that is presentonly in the high-resolution region.

Here, the integration control unit 41 c functions as an integrationprocessing unit that outputs an image recognition result integrated onthe basis of the reliability of the first image recognition result andthe reliability of the second image recognition result.

The integration control unit 41 c generates image data for displaying adesired image out of the entire image of each of the imaging units 21 to24, the panoramic synthesized image, and the like on a first displayunit 50, a second display unit 51, and the like. Also, the integrationcontrol unit 41 c generates a frame for displaying the recognized objectin an emphasized manner, CG for information, alerts, and the likeregarding the type, the size, the position, the speed, and the like ofthe target.

Also, display processing and the like for superimposing such CG andtexts on the image are performed. If the vehicle 1 is a police vehicle,for example, it is possible to recognize number plates of other vehiclesand a face image of a driver, to access a police server or the like viaa network for inquiry, and to display names of an owner and a driver ofthe vehicle on the first display unit 50, the second display unit 51,and the like. Here, at least one of the first display unit 50 and thesecond display unit 51 displays image data and displays the integratedimage recognition result.

In the first embodiment, the integration control unit 41 c is configuredto share information regarding the recognized object among the pluralityof camera units. For example, it is assumed that an object recognized bythe camera unit 14 has been recognized as moving in the direction of theangle of view of the camera unit 11. In such a case, the integrationcontrol unit 41 c transmits the type of the object and predictioninformation including information regarding the moving direction of theobject or priority recognition region information to the recognitionunit 31 b of the camera unit 11.

The recognition unit 31 b of the camera unit 11 performs a recognitionoperation with reference to the prediction information received from theintegration control unit 41 c. If the prediction information receivedfrom the integration control unit 41 c is information regarding themoving direction of the target, for example, which place in the angle ofview of the camera unit 11 the object will appear is predicted on thebasis of the information.

The integration control unit 41 c may transmit the priority recognitionregion information as the prediction information. The priorityrecognition region information includes which region in the imagingangle of view of the camera unit 11 the object will appear and when theobject will appear, for example. If the priority recognition regioninformation is received as the prediction information, the recognitionunit 31 b of the camera unit 11 predicts that there is a high likelihoodthat the object will appear in the priority recognition region at thepredicted timing and then performs image recognition.

The integration control unit 41 c performs communication with thetraveling control unit (ECU) 60 and the like in accordance with acommunication protocol based on CAN, FlexRay, Ethernet, or the like. Theintegration control unit 41 c thus performs display processing ofappropriately changing information to be displayed on the basis ofvehicle control signals from the traveling control unit (ECU) 60 and thelike. For example, the integration control unit 41 c changes a range ofan image to be displayed on the display unit in accordance with a movingstate of the vehicle 1 acquired by the vehicle control signals.

Note that the traveling control unit (ECU) 60 is a unit mounted in thevehicle 1 and incorporating a computer and a memory for comprehensivelyperforming driving control, direction control, and the like of thevehicle 1. Information regarding traveling (moving state) of the vehicle1 such as a traveling speed, a traveling direction, a shift lever, ashift gear, a direction indicator state, and an orientation of thevehicle 1 obtained by a geomagnetic sensor or the like, for example isinput as the vehicle control signals from the traveling control unit(ECU) 60 to the integration processing unit 40.

On the contrary, the integration control unit 41 c transmits informationsuch as the type, the position, the moving direction, the moving speed,and the like of the predetermined object (obstacle or the like)recognized by the recognition unit 41 b to the traveling control unit(ECU) 60. Thus, the traveling control unit (ECU) 60 performs controlnecessary to avoid the obstacle, such as stopping or driving of thevehicle 1 or a change in traveling direction. Here, the travelingcontrol unit (ECU) 60 functions as a mobile control unit that controlsmovement of the vehicle 1 on the basis of the integrated imagerecognition result.

The first display unit 50 is installed near the center of the frontupper portion of the driver’s seat of the vehicle 1 in the vehicle widthdirection with the display screen facing the rear side of the vehicle 1and functions as an electronic rear view mirror, for example. Note thatwhen a half mirror or the like is used and the first display unit 50 isnot used as a display, a configuration in which the first display unit50 can be used as a mirror may be adopted. For example, the firstdisplay unit 50 may include a touch panel or an operation button and isconfigured to be able to acquire an instruction from a user and outputthe instruction to the integration control unit 41 c.

The second display unit 51 is installed in the surroundings of anoperation panel near the center of the front side of the driver’s seatof the vehicle 1 in the vehicle width direction, for example. Note thata navigation system, an audio system, and the like are mounted in thevehicle 1 that is a mobile object.

Additionally, it is also possible to display various control signals andthe like from the navigation system, the audio system, and the travelingcontrol unit (ECU) 60 on the second display unit, for example. Forexample, the second display unit includes a touch panel or an operationbutton and is configured to be able to acquire an instruction from theuser.

In the first embodiment, the first display unit 50 and the seconddisplay unit 51 include liquid crystal displays or organic EL displaysas display panels. Note that in the first embodiment, the number ofdisplay units is not limited to two. One display unit may be used, orthree or more display units may be used.

In the first embodiment, some or all of the components included in theintegration processing unit 40 may be realized by hardware or may berealized by the CPU 42 being caused to execute the computer programstored in the memory 43. As the hardware, it is possible to use adedicated circuit (ASIC), a processor (reconfigurable processor, DSP),or the like.

In the first embodiment, a part or entirety of the image processingperformed by the image processing units 31 a to 34 a may be performed bythe image processing unit 41 a of the integration processing unit 40. Inthe first embodiment, the image acquisition unit and the first imagerecognition unit are accommodated in the casing of the same camera unit,for example, and the camera unit and the second image recognition unitare accommodated in different casings.

Although the integration processing unit 40 is mounted in the vehicle 1that is a mobile object in the first embodiment, a part of processing ofthe image processing unit 41 a, the recognition unit 41 b, and theintegration control unit 41 c in the integration processing unit 40 maybe performed by an external server or the like via a network, forexample.

In such a case, although the imaging units 21 to 24 that are imageacquisition units are mounted in the vehicle 1, for example, some of thefunctions of the camera processing units 31 to 34 and the integrationprocessing unit 40 can be processed by an external device such as anexternal server or the like, for example.

A recording unit 61 records the entire image of each of the imagingunits 21 to 24 generated by the integration processing unit 40 and thepanoramic synthesized image in a recording medium. Moreover, therecording unit 61 records CG of a predetermined frame for indicating therecognized target, texts, alerts, and the like and images displayed onthe first display unit 50, the second display unit 51, and the like withCG superimposed thereon along with a clock time, GPS, information andthe like.

The integration processing unit 40 can also reproduce past informationrecorded in the recording unit 61 and display it on the first displayunit 50 and the second display unit 51.

A communication unit 62 is for communicating with the external serverand the like via a network and can transmit information before beingrecorded in the recording unit 61 and past information recorded in therecording unit 61 to the external server and the like and save them inthe external server and the like.

On the contrary, it is also possible to acquire congestion informationand various kinds of information from the external server and the likeand to display the information on the first display unit 50 and thesecond display unit 51 via the integration processing unit 40.

As described above, the image recognition processing is performed on thehigh-resolution region 10 a in the state of the image data beforedistortion correction. This enables quick recognition, and it ispossible to improve recognition accuracy of the image data beforedistortion correction obtained from the low-resolution region 10 b byperforming the image recognition processing after the distortioncorrection.

Furthermore, since the image recognition processing is performed by therecognition unit of the imaging device on the high-resolution region 10a by using hardware in the state of the image data before the distortioncorrection, it is possible to more quickly perform the image recognitionof the high-resolution region 10 a in the first embodiment.

Note that a part or entirety of the image recognition processingperformed by the recognition units 31 b to 34 b may be performed by therecognition unit 41 b of the integration processing unit 40. However, insuch a case, the recognition unit 41 b can quickly obtain the imagerecognition result in a short period of time by performing the imagerecognition processing on the high-resolution region 10 a in the stateof the image data before the distortion correction. On the other hand,the image recognition processing is performed on the image data beforethe distortion correction obtained from the low-resolution region 10 bafter the distortion correction.

Note that the image processing performed by the recognition units 31 bto 34 b or the recognition unit 41 b may be performed using machinelearning. As a recognition method using machine learning, anyrecognition method may be used as long as the recognition method uses anobject detection algorithm.

For example, it is possible to use You Only Look Once (YOLO),Region-based Convolution Neural Networks (R-CNN), Fast R-CNN, FasterR-CNN, Single Shot MultiBox Detector (SSD), or the like. Note that it ispossible to enhance an image recognition rate even if the distortedimage is used as it is to some extent, by using machine learning.

FIG. 4 is a flowchart for explaining a processing flow of the cameraprocessing unit in the first embodiment. The processing flow in FIG. 4is controlled in units of frames, for example, by the hardware inconjunction with the CPU, the GPU, and the like by the CPUs inside thecamera processing units 31 to 34 executing computer programs in thememories.

Once the image processing system 100 is turned on, the hardware isreset, and the flow is started. Thereafter, the flow in FIG. 4 isexecuted every time an orthogonal synchronous signal is input, and thecamera processing units 31 to 34 acquire captured images by using theimaging units 21 to 24, respectively, in Step S41 (an imaging step or anacquisition step).

In Step S42, the image processing units 31 a to 34 a inside the cameraprocessing units 31 to 34 perform image processing such as De-Bayerprocessing and white balance adjustment and generate image data beforedistortion correction.

In Step S43 (first image recognition step), the recognition units 31 bto 34 b perform image recognition of a predetermined object from atleast the high-resolution region 10 a on the basis of image data beforedistortion correction. In Step S43, image recognition is performed onimage data in at least a partial region including a low-distortionregion out of the image data acquired in Step S41, and a first imagerecognition result is output.

Note that the recognition units 31 b to 34 b may perform imagerecognition on the predetermined object only from the high-resolutionregion 10 a as described above or may perform image recognition on theimage in the surrounding low-resolution region 10 b as well.

In Step S44, the type and the coordinates (or information of the regionof the target) of the object after image recognition are transmitted asa set to the integration processing unit 40.

FIG. 5 is a flowchart for explaining a processing flow performed by theintegration processing unit 40 in the first embodiment. The processingflow in FIG. 5 is controlled by the CPU 42 of the integration processingunit 40 executing the computer program in the memory 43.

In Step S51, the integration processing unit 40 acquires camerainformation of each of the camera information units 31 c to 34 c of thecamera processing units 31 to 34.

In Step S52, a distortion correction parameter is calculated on thebasis of optical properties in the camera information, the number ofpixels of the imaging devices, and the like. Note that a coordinateconversion table may be prepared in advance instead of calculating thedistortion correction parameter, and the distortion correction may beperformed using the coordinate conversion table.

Also, interpolation may be performed at the time of the distortioncorrection. Also, a synthesis parameter for synthesizing images from theplurality of camera units is also calculated on the basis of attachmentcoordinates and posture information in the vehicle coordinates of eachcamera unit in the camera information in Step S52.

In Step S53, the distortion correction parameter calculated in Step S52and the synthesis parameter are set in the image processing unit 41 ainside the integration processing unit 40.

In Step S54, a coordinate conversion expression for positioning thecoordinates of the low-resolution region after distortion correction andthe coordinates of the high-resolution region with no distortioncorrection performed thereon is calculated. Also, coordinate conversionexpressions for positioning coordinates when the images from theplurality of camera units are synthesized are calculated, and thesecoordinate conversion expressions are set in the integration controlunit 41 c. In addition, interpolation may be performed in thesecoordinate conversion expressions.

In Step S55, an image processing parameter is generated on the basis ofsensitivity properties, gamma properties, and the like of the imagingdevices in the camera information. At that time, the image processingparameter may be generated such that an image recognition rate isimproved by statistically processing the image data before thedistortion correction.

In Step S56, the image processing parameter generated in Step S55 is setin the image processing unit 41 a inside the integration processing unit40.

FIG. 6 is a flowchart for explaining an integration processing flowperformed by the integration processing unit 40 in the first embodiment.The processing flow in FIG. 6 is controlled in units of frames, forexample, by the hardware in conjunction with the CPU, the GPU, and thelike by the CPU 42 of the integration processing unit 40 executing thecomputer program in the memory 43.

In Step S61, the integration processing unit 40 acquires the image databefore the distortion correction by using the camera processing units 31to 34.

In Step S62, the image processing unit 41 a of the integrationprocessing unit 40 performs image processing on the image data beforethe distortion correction, performs distortion correction, and alsosynthesizes the images from the plurality of camera units. As thedistortion correction parameter and the synthesis parameter at thattime, the parameters set in Step S53 are used. Note that Step S62functions as the synthesis step of synthesizing the plurality of piecesof image data captured by the plurality of camera units.

In Step S63 (second image recognition step), the recognition unit 41 bof the integration processing unit 40 performs image recognition onimage data in the low-resolution region (high-distortion region) afterthe distortion correction and the other image data. In the second imagerecognition step, image recognition is performed on image data in awider region than the partial region on which recognition is performedin Step S43 in the first image recognition step out of image dataacquired in Step S41 that is the acquisition step.

Also, image recognition is performed on an object which lies across aplurality of pieces of image data obtained from the plurality of cameraunits by performing image recognition on the synthesized image obtainedby synthesizing the image data from the plurality of camera units aswell. Furthermore, image recognition is also performed on the objectwhich moves across the plurality of camera units.

In Step S64, the recognition unit 41 b of the integration processingunit 40 acquires a result (the recognized object and the coordinatesthereof) of the image recognition performed by the recognition units 31b to 34 b of the camera processing units 31 to 34 on each piece of imagedata before the distortion correction. Note that Steps S64 and S65 areperformed in parallel with Steps S61 to S63.

In Step S65, the coordinates of the object recognized by the recognitionunits 31 b to 34 b of the camera processing units 31 to 34 are convertedinto coordinates after the distortion correction.

In Step S66 (integration processing step), the image recognition resultsof the recognition units 31 b to 34 b of the camera processing units 31to 34 and the image recognition result of the recognition unit 41 b ofthe integration processing unit 40 are compared with each other, and afinal recognition result is generated.

At this time, the final recognition result is generated on the basis ofreliability of each of the image recognition results as described above.If the recognition results of the recognition units 31 b to 34 b aredifferent from the recognition result of the recognition unit 41 b, forexample, a recognition result with higher reliability is adopted.

In Step S67, a frame for displaying the object after image recognitionin an emphasized manner is generated, and it is superimposed on theimage after the distortion correction.

In Step S68, the image is displayed with the frame superimposed thereonon the first display unit 50, the second display unit 51, or the like.At that time, the display region of the image displayed on the firstdisplay unit 50, the second display unit 51, or the like and therecognition region of the recognition units 31 b to 34 b and 41 b arechanged in accordance with the moving state of the vehicle 1. Detailsthereof will be described later.

In Step S69, the coordinates of the object after image recognition withrespect to the coordinates of the vehicle 1 are generated. Note thatSteps S69 and S70 are performed in parallel with Steps S67 and S68.

In Step S70, the coordinates of the object with respect to the vehicle 1generated in Step S69 and the type of the object are transmitted to thetraveling control unit (ECU) 60.

In Step S71, if there is an object moving between camera units, themoving direction and the speed of the object are predicted. For example,the integration control unit 41 c recognizes an object that moves acrossa plurality of pieces of image data and predicts a motion thereof. Notethat Steps S71 and S72 are performed in parallel with Steps S69 and S70.

In Step S72, the type of the object and information regarding the movingdirection or the like or prediction information regarding a priorityrecognition region are transmitted to the corresponding camera unit. Inthis manner, it is possible to improve accuracy of the image recognitionon the object on the side of the camera unit. Thereafter, the flowchartin FIG. 6 is ended.

Note that the flowchart in FIG. 6 is executed in units of frames, forexample. Also, the parallel processing in FIG. 6 may be realized byhardware processing at least one process of the parallel processing.

FIG. 7 is a flowchart for explaining an example of an image displaymethod in the first embodiment. FIG. 7 explains details of Step S68 inFIG. 6 , and the processing flow in FIG. 7 is controlled in units offrames, for example, by the CPU 42 of the integration processing unit 40executing the computer program in the memory 43.

FIG. 8 is a diagram for explaining a relationship of a high-resolutionregion, a low-resolution region, and a plurality of types of displayregions in the first embodiment. FIG. 8(A) is a diagram for explainingan example of a display region 82 with a reference angle of view of thecamera units 11, 12, and 14, and FIG. 8(B) is a diagram for explainingan example of a display region 83 with a narrow angle of view of thecamera units 11, 12, and 14.

FIG. 8(C) is a diagram for explaining an example of a display region 84with a wide angle of view of the camera units 11, 12, and 14, and FIG.8(D) is a diagram for explaining an example of a display region 85 witha narrow angle of view behind the camera unit 13. Also, FIG. 8(E) is adiagram for explaining an example of a display region 86 with a wideangle of view behind the camera unit 13.

In FIGS. 8(A) to (E), 81 denotes the light receiving surface of theimaging device, 10 a denotes the high-resolution region (low-distortionregion) described in FIGS. 2, and 10 b denotes the low-resolution region(high-distortion region). However, the boundary between the abovehigh-resolution region (low-distortion region) 10 a and thelow-resolution region (high-distortion region) 10 b is not displayed inthe image that is ordinarily displayed.

However, the aforementioned boundary may be displayed in a superimposedmanner on the image as needed. In the first embodiment, the regionrecognized by the recognition unit 41 b is the entire display region,for example. Also, the region recognized by the recognition units 31 bto 34 b has, for example, rectangular shapes that are inscribed in thehigh-resolution region 10 a in the display region. As illustrated inFIGS. 8(A) to (E), the plurality of types of display regions can beswitched, and these display regions are switched on the basis of vehiclecontrol information from the traveling control unit (ECU) 60.

In Step S73 in FIG. 7 , the CPU 42 acquires the vehicle controlinformation from the traveling control unit (ECU) 60. The vehiclecontrol information includes, for example information regardingtraveling of the vehicle 1 such as a traveling speed, a travelingdirection, a shift lever, a shift gear, and a direction indicator, forexample, as described above.

In Step S74, the CPU 42 determines whether or not the vehicle 1 is in aforward traveling state on the basis of the vehicle control information.If it is determined that the vehicle 1 is in the forward traveling state(Yes in Step S74), the CPU 42 moves on to Step S75. If it is determinedthat the vehicle 1 is not in the forward traveling state (No in StepS74), the CPU 42 moves on to Step S80.

In Step S75, the CPU 42 determines whether or not a route is beingchanged. If it is determined that the route is being changed (Yes inStep S75), the CPU 42 moves on to Step S76. If it is determined that theroute is notbeing changed (No in Step S75), the CPU 42 moves on to StepS79.

In Step S76, the CPU 42 determines whether or not the forward travelingspeed is greater than a predetermined threshold value V1. If it isdetermined that the forward traveling speed is greater than thepredetermined threshold value V1 (Yes in Step S76), the CPU 42 moves onto Step S78. If it is determined that the forward traveling speed is notgreater than the predetermined threshold value V1 (No in Step S76), theCPU 42 moves on to Step S77.

In Step S77, the CPU 42 causes the first display unit 50 to display animage with a reference angle of view. For example, the CPU 42 causes thefirst display unit 50 to display the image in the display region 82 withthe reference angle of view in FIG. 8 .

FIG. 9A, FIG. 9B, and FIG. 9C are diagrams for explaining displayexamples of the first display unit 50. FIG. 9A is a diagram forexplaining a display example of an image with a reference angle of view,FIG. 9B is a diagram for explaining a display example of an image with anarrow angle of view, and FIG. 9C is a diagram for explaining a displayexample of an image with a wide angle of view.

In FIGS. 9A, 501 denotes a display screen of the first display unit 50.82R denotes an image with a reference angle of view of the camera unit14 on the left side, 82C denotes an image with a reference angle of viewof the camera unit 11 at the front, and 82R denotes an image with areference angle of view of the camera unit 12 on the right side.

If the image in the display region 82 with the reference angle of viewin FIG. 8 is displayed on the first display unit 50, for example, theimage is displayed as in FIG. 9A, for example. For example, an image 82Lwith a reference angle of view of the camera unit 14 on the left side,an image 82C with a reference angle of view of the camera unit 11 at thefront, and an image 82R with a reference angle of view of the cameraunit 12 on the right side are aligned in this order from the left sideand are displayed on the display screen 501 of the first display unit50.

FIG. 10A is a diagram for explaining a display example of the displayscreen 501 of the first display unit 50 in the first embodiment. FIG.10B is a diagram for explaining a display example of the display screen511 of the second display unit 51 in the first embodiment.

As illustrated in FIG. 10A, if it is determined that the forwardtraveling speed is not greater than the predetermined threshold value V1in Step S76, then the images 82L, 82C, and 82R with the reference angleof view are aligned and displayed on the display screen 501 of the firstdisplay unit 50.

Note that in FIG. 10A, 82B is an image with a reference angle of view ofthe camera unit 13 for the rear side and is displayed as a picture in apicture on the display screen 501 of the first display unit 50. In thefirst embodiment, if it is determined that the forward traveling speedis not greater than the predetermined threshold value V1 in Step S76,the image 82B with the reference angle of view of the camera unit 13 isdisplayed on the first display unit 50.

In Step S78, the CPU 42 causes the first display unit 50 to display theimage in the display region 83 with the narrow angle of view illustratedin FIG. 8(B). The display region 83 is wider on the upper side and has anarrower width in the left-right direction as compared with the displayregion 82. Also, an image 83L with a narrow angle of view of the cameraunit 14 on the left side, an image 83C with a narrow angle of view ofthe camera unit 11 at the front, and an image 83R with a narrow angle ofview of the camera unit 12 on the right side are displayed in an alignedmanner as in FIG. 9B.

Since the view becomes narrow if the forward traveling speed is greaterthan the predetermined threshold value V1 (60 km, for example) in thismanner, necessary information can be easily and quickly viewed if thedisplay as in FIG. 9B is performed.

In Step S79, the CPU 42 causes the first display unit 50 to display theimage in the display region 84 with the wide angle of view illustratedin FIG. 8(C). The display region 84 has a wider width in the left-rightdirection and is widened in the lower direction as compared with thedisplay region 82. As illustrated in FIG. 9C, for example, an image 84Lwith a wide angle of view of the camera unit 14 on the left side, animage 84C with a wide angle of view of the camera unit 11 at the front,and an image 84R with a wide angle of view of the camera unit 12 on theright side are aligned and displayed on the display screen 501 of thefirst display unit 50.

Furthermore, if the route is being changed to the left side, forexample, the three images displayed in an aligned manner in FIG. 9C maybe displayed with leftward deviation with respect to the center of thedisplay screen 501. On the contrary, if the route is being changed tothe right side, the three images displayed in an aligned manner in FIG.9C may be displayed with rightward deviation with respect to the centerof the display screen 501.

Such display can enhance visibility. Since the images with the wideangles of view are displayed when the route is being changed, it ispossible to more easily view safety of the surroundings. Furthermore,since the image with an angle of view widened on the lower side isdisplayed, it is possible to more easily view an obstacle on a road.

In Step S80, the CPU 42 determines whether or not the backward travelingspeed is greater than a predetermined speed V2 (10 km, for example). Ifit is determined that the backward traveling speed is greater than thepredetermined speed V2 (Yes in Step S80), the CPU 42 moves on to StepS82. If it is determined that the backward traveling speed is notgreater than the predetermined speed V2 (No in Step S80), the CPU 42moves on to Step S81.

In Step S81, the CPU 42 causes the display screen 511 of the seconddisplay unit 51 to display an image (the image as illustrated in FIG.8(D)) in the display region 85 with a narrow angle of view in theup-down direction for backward traveling as in FIG. 10B. FIG. 10Billustrates an example of a screen displayed on the display screen 511of the second display unit 51 when the vehicle 1 moves backward, and aguide 512 for guiding the vehicle 1 to a parking space is displayed in asuperimposed manner, for example.

FIG. 11A is a diagram for explaining an example of the image 85B with anarrow angle of view in the up-down direction at the time of backwardmoving, and the image 85B with the narrow angle of view in the up-downdirection as in FIG. 11A is displayed on the display screen 511 of thesecond display unit 51.

In Step S82, the CPU 42 causes the display screen 511 of the seconddisplay unit 51 to display the image 86B with a wide angle of view inthe up-down direction for backward traveling (such as an imageillustrated in FIG. 8(E)) as in FIG. 11B. Here, FIG. 11B is a diagramillustrating an example of the image 86B with a wide angle of view forbackward traveling in the first embodiment.

As illustrated in FIG. 8(E), the display region 86 with a wide angle ofview for backward traveling has an angle of view widened in the up-downdirection as compared with the display region 85 with a narrow angle ofview for backward traveling. This is for easy viewing of an obstaclethrough further backward display if the backward traveling speed isgreater than the predetermined speed V2.

Note that although the left-right width of the display region 86 with awide angle of view for backward traveling is the same as the left-rightwidth of the display region 85 with a narrow angle of view for backwardtraveling in the first embodiment, the left-right width of the displayregion 86 may be narrower than the left-right width of the displayregion 85.

In this manner, the high-resolution region (low-distortion region) 10 ais configured to have projection properties approximated to the centerprojection method (y = f × tanθ) or the equidistant projection method (y= f × θ) of the optical system for normal imaging as described above inthe first embodiment.

Therefore, an image for an electronic rear view mirror, for example,displayed on the first display unit 50 has a higher resolution ascompared with the low-resolution region (high-distortion region) 10 band can display further locations on the front side, the lateral sides,and the rear side of the vehicle 1 with higher definition.

Also, since the high-resolution region 10 a includes small opticaldistortion, it is also possible to display the image for the electronicrear view mirror displayed on the first display unit 50 in a state withsmall distortion, and the driver can view the surroundings of thevehicle 1 with more natural perspective.

Moreover, since the image recognition is performed on thehigh-resolution region 10 a in the state of the image data beforedistortion correction, it is possible to set an image recognition timingearlier when image recognition is performed for a number plate of asurrounding vehicle, a person, an obstacle and the like and to enhanceimage recognition accuracy.

Since the high-resolution region 10 a in the first embodiment isconfigured to include small optical distortion, and it is possible toperform image recognition in the state of image data before distortioncorrection, it is possible to reduce a processing load for the imagerecognition and to perform the image recognition at a high speed. It isthus possible to discover an obstacle in an early stage on the basis ofthe image recognition result and to timely perform an operation foravoiding the obstacle.

If the configuration in the first embodiment is used in this manner, itis possible to obtain great effects at the time of high-speed travelingalong a highway, for example. Note that although the example in whichthe plurality of camera units are used has been described in the firstembodiment, the configuration is also effective in a system includingonly one camera unit.

As described above, high-definition display for an electronic rear viewmirror and wide-range display for checking the surroundings, such as arear side, of a vehicle are acquired at the same time with a smallnumber of cameras, and an extracted region of an image to be output ischanged in accordance with a vehicle control state, in the firstembodiment. Also, the display angle of view is changed on the basis oftraveling speed information (including the moving state of the vehicle1) of the vehicle 1. It is thus possible for the driver to easily checka more important range in accordance with the vehicle control state.

Moreover, the image recognition region is changed by changing theextracted region in accordance with the moving state of the vehicle 1.For example, the display region of the image to be displayed on thedisplay unit and the recognition region of the first image recognitionunit and the second image recognition unit are changed by changing theextracted region in accordance with the moving state of the vehicle 1,and it is thus possible to perform efficient image recognition with lesswaste. Note that the display region and the recognition region may notbe the same.

Note that not only the extracted region may be changed but also theresolution may be changed in the first embodiment. For example, thedisplay angle of view may be narrowed, and the resolution of thesurrounding angle of view may be lowered at the time of high-speedtraveling.

Although the case in which speed information is used as the vehiclecontrol state has been described as an example in the first embodiment,obstacle information in the surroundings of the vehicle may be acquiredfrom the imaging units 21 to 24 or other sensors, for example, and thedisplay region may be changed on the basis of the obstacle information.

Note that the example in which the image processing system is mounted ina mobile object such as the vehicle 1 has been described in the firstembodiment. However, the mobile object in the first embodiment is notlimited to a vehicle such as an automobile and may be any mobile objectsuch as a train, a ship, an aircraft, a robot, or a drone as long as themobile object moves.

Also, the image processing system in the first embodiment includes anyimage processing system mounted in such a mobile object. Additionally,it is also possible to apply the first embodiment to a case in which themobile object is remotely controlled.

Note that the example in which the imaging units 21 to 24 are used asimage acquisition units has been described in the first embodiment.However, the image acquisition unit may be any image acquisition unit aslong as it acquires image data generated by imaging devices that captureoptical images including a low-distortion region and a high-distortionregion and may be an image acquisition unit that acquires image data asdescribed above via a network, for example. Alternatively, the imageacquisition unit may acquire the image data as described above recordedin the recording medium by reproducing it.

Second Embodiment

At least one of various functions, processing, and methods describedabove in the first embodiment can be realized using a program.Hereinafter, the program for realizing the at least one of the variousfunctions, processing, and the methods described above in the firstembodiment will be referred to as a “program X” in a second embodiment.

Moreover, a computer for executing the program X will be referred to asa “computer Y” in the second embodiment. A personal computer, amicrocomputer, a central processing unit (CPU), or the like is anexample of the computer Y. The computer such as the image processingsystem in the aforementioned embodiment is also an example of thecomputer Y.

At least one of the various functions, processing, and the methodsdescribed above in the first embodiment can be realized by the computerY executing the program X. In this case, the program X is supplied tothe computer Y via a computer-readable storage medium.

The computer-readable storage medium in the second embodiment includesat least one of a hard disk device, a magnetic storage device, anoptical storage device, a photomagnetic storage device, a memory card, aROM, a RAM, and the like. Moreover, the computer-readable storage mediumin the second embodiment is a non-transitory storage medium.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Pat. App. No.2021-155814 filed on Sep. 24, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing system comprising at leastone processor or circuit configured to function as: an image acquisitionunit configured to acquire image data generated by an imaging devicethat captures an optical image including a low-distortion region and ahigh-distortion region; a first image recognition unit configured toperform image recognition on image data of at least a partial region inthe image data acquired by the image acquisition unit to output a firstimage recognition result; a second image recognition unit configured toperform image recognition on image data of a region wider than thepartial region in the image data acquired by the image acquisition unitto output a second image recognition result; and an integrationprocessing unit configured to output an image recognition resultintegrated on the basis of the first image recognition result and thesecond image recognition result.
 2. The image processing systemaccording to claim 1, wherein the partial region is a regioncorresponding to the low-distortion region.
 3. The image processingsystem according to claim 1, wherein the second image recognition unitperforms image recognition on image data corresponding to thelow-distortion region and the high-distortion region and outputs thesecond image recognition result.
 4. The image processing systemaccording to claim 1, wherein the image acquisition unit includes anoptical system that forms the optical image, and an imaging device thatcaptures the optical image generated by the optical system.
 5. The imageprocessing system according to claim 4, wherein the image acquisitionunit and the first image recognition unit are accommodated in the samehousing of a camera unit.
 6. The image processing system according toclaim 5, wherein the camera unit and the second image recognition unitare accommodated in separate housings.
 7. The image processing systemaccording to claim 1, wherein the first image recognition unit performsimage recognition on image data corresponding to the low-distortionregion without distortion correction and outputs the first imagerecognition result.
 8. The image processing system according to claim 1,wherein the first image recognition unit performs image recognition onimage data corresponding to the low-distortion region in a state beforedistortion correction and outputs the first image recognition result. 9.The image processing system according to claim 1, wherein the secondimage recognition unit performs image recognition on image datacorresponding to the high-distortion region in a state after distortioncorrection is performed on the high-distortion region and outputs thesecond image recognition result.
 10. The image processing systemaccording to claim 1, wherein the integration processing unit outputsthe integrated image recognition result on the basis of reliability ofthe first image recognition result and reliability of the second imagerecognition result.
 11. The image processing system according to claim1, further comprising at least one processor or circuit configured tofunction as: a display unit configured to display the image dataacquired by the image acquisition unit and the integrated imagerecognition result.
 12. The image processing system according to claim11, wherein the image acquisition unit is mounted in a mobile object,and the integration processing unit changes a display region of an imagedisplayed on the display unit and a recognition region recognized by thefirst image recognition unit and the second image recognition unit inaccordance with a moving state of the mobile object.
 13. The imageprocessing system according to claim 4, wherein when a focal distance ofthe optical systems is defined as f, a half angle of view is defined asθ, an image height on an image plane is defined as y, and a projectionproperty representing a relationship between the image height y and thehalf angle of view θ is defined as y(θ), y(θ) in the low-distortionregion is greater than f × θ and is different from the projectionproperty in the high-distortion region.
 14. The image processing systemaccording to claim 13, wherein the low-distortion region is configuredto have a projection property that is approximated to a centerprojection method (y = f × tanθ) or an equidistant projection method (y= f × θ).
 15. The image processing system according to claim 13, whereinwhen θmax is defined as a maximum half angle of view that the opticalsystem has and A is defined as a predetermined constant, the imageprocessing system is configured to satisfy$1 < \frac{f \times \sin\theta_{\max}}{y\left( \theta_{\max} \right)} \leq A$.
 16. A mobile object controlled by an image processing system, whereinthe image processing system includes at least one processor or circuitconfigured to function as: an image acquisition unit configured toacquire image data generated by an imaging device that captures anoptical image having a low-distortion region and a high-distortionregion; a first image recognition unit configured to perform imagerecognition on image data of at least a partial region in the image dataacquired by the image acquisition unit to output a first imagerecognition result; a second image recognition unit configured toperform image recognition on image data of a region wider than thepartial region in the image data acquired by the image acquisition unitto output a second image recognition result; and an integrationprocessing unit configured to output an image recognition resultintegrated on the basis of the first image recognition result and thesecond image recognition result, and wherein the mobile object comprisesat least one processor or circuit configured to function as a travelingcontrol unit configured to control traveling of the mobile object on thebasis of the integrated image recognition result.
 17. An imageprocessing method comprising: an acquisition step of acquiring imagedata generated by an imaging device that captures an optical imagehaving a low-distortion region and a high-distortion region; a firstimage recognition step of performing image recognition on image data ofat least a partial region in the image data acquired in the acquisitionstep to output a first image recognition result; a second imagerecognition step of performing image recognition on image data of aregion wider than the partial region in the image data acquired in theacquisition step to output a second image recognition result; and anintegration processing step of outputting an image recognition resultintegrated on the basis of the first image recognition result and thesecond image recognition result.
 18. A non-transitory computer-readablestorage medium configured to store a computer program comprisinginstructions for executing the following processes: an acquisition stepof acquiring image data generated by an imaging device that captures anoptical image having a low-distortion region and a high-distortionregion; a first image recognition step of performing image recognitionon image data of at least a partial region in the image data acquired inthe acquisition step to output a first image recognition result; asecond image recognition step of performing image recognition on imagedata of a region wider than the partial region in the image dataacquired in the acquisition step to output a second image recognitionresult; and an integration processing step of outputting an imagerecognition result integrated on the basis of the first imagerecognition result and the second image recognition result.